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Abstract. Generating functions of plane overpartitions are ob- 
tained using various methods: non-intersecting paths, RSK type 
algorithms and symmetric functions. We extend some of the results 
to cylindric partitions. Also, we show that plane overpartitions cor- 
respond to domino tilings and we give some basic properties of this 
correspondence. 



1. Introduction 

The goal of the first part of this paper is to introduce a new object: 
the plane overpartitions and to give several enumeration formulas for 
these plane overpartitions. A plane overpartition is a plane partition 
where in each row the last occurrence of an integer can be overlined 
or not and in each column the first occurrence of an integer can be 
overlined or not and all the others are overlined. An example of a 
plane overpartition is 

4 4 4 3 
4 3 3 3 
4 3 
3 

This is an overpartition of the shape (4, 4, 2, 1), with the weight equal 
to 38 and 6 overlined parts. 

This paper takes its place in the series of papers on overpartitions 
started by Corteel and Lovejoy [CL] . The motivation is to show that 
the generating function of plane overpartitions is : 



n 

n>l 



In this paper, we give several proofs of this result and several refine- 
ments and generalizations. Namely, we give 



Result 1. the hook-length formula for the generating function of plane 
overpartitions of a given shape, see Theorem\^ 
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Result 2. the hook formula for the generating function for reverse 
plane overpartitions, see Theorem\^ 

Result 3. the generating formula for plane overpartitions with bounded 
parts, see Theorem\^ 

The goal of the second part of this paper is to extend the generating 
formula for cylindric partitions due to Borodin and the 1-parameter 
generalized MacMahon's formula due to Vuletic: 

(1.1) Y. -^nw^^-nf^^)". 

n is a n=l ^ ^ 

plane partition 

where the weight An(t) is a polynomial in t that we describe below. 

Given a plane partition 11, we decompose each connected component 
(a connected set of boxes filled with a same number) of its diagram 
into border components (i.e. rim hooks) and assign to each border 
component a level which is its diagonal distance to the end of the 
component. We associate to each border component of level i, the 
weight (1 — t*). The weight of the plane partition 11 is Ajiit) the 
product of the weights of its border components. See [V2j for further 
details and Figure [T] for an example of a plane partition of weight 

(l-tl0)(l-t2)2(l_t3)2_ 




Figure 1 . Weight of a plane partition 

We give a new proof of the 1-parameter generalized MacMahon's 
formula. We also extend this formula to two more general objects: 
skew plane partitions and cylindric partitions. We give 

Result 4. 1-parameter generalized formula for skew plane partitions, 
see Theorem^ 

Result 5. 1-parameter generalized formula for cylindric partitions, see 
Theorem 0. 
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In the rest of this section we give definitions and explain our results 
in more detail. 

A partition A is a nonincreasing sequence of positive integers (Ai, . . . , A^). 
The AjS are called parts of the partition and the number of parts is de- 
noted by -^(A). The weight |A| of A is the sum of its parts. A partition A 
can be graphically represented by the Ferrers diagram that is a diagram 
formed of i{X) left justified rows, where the i*^ row consists of Aj cells 
(or boxes). The conjugate of a partition A, denoted with A', is a parti- 
tion that has the Ferrers diagram equal to the transpose of the Ferrer 
diagram of A. For a cell of the Ferrers diagram of A the hook 
length of this cell is hij = Xi + X'j—i—j + 1 and the content Cjj = j — i. 
It is well known that the generating function of partitions that have at 
most n parts is l/(g)„,, where (a)„ = (a; g)„ = 11^=0^(1 ~ ^Q'')- More 
definitions on partitions can be found, for example, in [X] or [Mac]. 

An overpartition is a partition where the last occurrence of an in- 
teger can be overlined |CLj . Last occurrences in an overpartition are 
in one-to-one correspondence with corners of the Ferrers diagram and 
overlined parts can be represented by marking the corresponding cor- 
ners. The generating function of overpartitions that have at most n 
parts is (-<?)„/(<?)„. 

Let A be a partition. A plane partition of shape A is a filling of 
cells of the Ferrers diagram of A with positive integers that form a 
nonincreasing sequence along each row and each column. We denote 
the shape of a plane partition 11 with sh(n) and the sum of all entries 
with |n|, called the weight of 11. It is well known, under the name of 
MacMahon's formula, that the generating function of plane partitions 
is 



One way to prove this is to construct a bijection between plane par- 
titions and pairs of semi-standard reverse Young tableaux of a same 
shape and to use the inverse of the RSK algorithm, that gives a bijec- 
tion between those pairs of those tableaux [BKj . 

Recall that a plane overpartition is a plane partition where in each 
row the last occurrence of an integer can be overlined or not and in 
each column the first occurrence of an integer can be overlined or not 
and all the others are overlined. This definition implies that the entries 
strictly decrease along diagonals. Therefore a plane overpartition is a 
plane partition made of rim hooks (connected skew shapes containing 
no 2 X 2 block of squares) where some entries are overlined. More 



n is a 
plane partition 
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precisely, it is easy to check that inside a rim hook only one entry can 
be chosen to be overlined or not and this entry is the upper right entry. 

Plane overpartitions are therefore in bijection with rim hook plane 
partitions where each rim hook can be overlined or not (or weighted 
by 2). Recently, those weighted rim hook plane partitions were stud- 
ied in jFWl iFWn IVH |V2]. The first result obtained was the shifted 
MacMahon's formula that says that the generating function of plane 
overpartitions is 

n is a n=l ^ ^ ^ 

plane overpartition 

This was obtained as a limiting case of the generating formula for plane 
overpartitions which fit into a r x c box, i.e. whose shapes are contained 
in the rectangular shape with r rows and c columns. 

Theorem 1. |FWl IVl] The generating function of plane overpartitions 
which fit in a r X c box is 

n n 1 _ fli+i-i 
i=i j=i ^ 

This theorem was proved in |FWl IVlj using Schur P and Q symmet- 
ric functions and a suitable Fock space. In jV2j the theorem was proved 
in a bijective way where a RSK-type algorithm (due to Sagan [Sa], see 
also Chapter XIII of |HHj ) was used to construct a bijection between 
plane overpartitions and matrices of nonnegative integers where posi- 
tive entries can be overlined. 



Note that plane overpartitions are also in bijection with super semi- 
standard tableaux |KrOj using the jeu de taquin |Savj . 

In Section 2, we give a mostly combinatorial proof of the generalized 
MacMahon formula |V2j . Namely, we prove 

Theorem 2. [V2] 

Ue'P{r,c) i=l j=l 



where V{r, c) is the set of plane partitions with at most r rows and 
c columns. When we set t = —1, only the border components of level 
1 have a non-zero weight and we get back Theorem [H 
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The main result of Section 3 is a hook-content formula for the gen- 
erating function of plane overpartitions of a given shape. More gener- 
ally, we give a weighted generating function where overlined parts are 
weighted by some parameter a. Throughout Sections 3, 4 and 5 we 
assume these weights, unless otherwise stated. 

Let S{X) be the set of all plane overpartitions of shape A. The 
number of overlined parts of an overpartition 11 is denoted with o(n). 

Theorem 3. Let X be a partition. The weighted generating function 
of plane overpartitions of shape A is 



We also give the weighted generating formula for plane overpartitions 
"bounded" by A, where by that we mean plane overpartitions such that 
the i^^ row of the plane overpartition is an overpartition that has at 
most Xi parts and at least Aj+i parts. Let B{X) be the set of all such 
plane overpartitions. 

Theorem 4. Let X be a partition. The weighted generating function 
of plane overpartitions such that the i^^ row of the plane overpartition 
is an overpartition that has at most Aj parts and at least Aj+i parts is 



Cij = j — i is the content of the cell 

Note that it is enough to assign weights to overlined (or nonoverlined) 
parts only because generating functions where overlined and nonover- 
lined parts are weighted by a and b, respectively, follow trivially from 
the above formulas. 

We prove these two theorems using a correspondence between plane 
overpartitions and sets of nonintersecting paths that use three kinds of 
steps. The work of Brenti used similar paths to compute super Schur 
functions |Brj . Using Gessel-Viennot results [GV] we obtain determi- 
nantal formulas for the generating functions above. Then we evaluate 
the determinants to obtain the hook-content formulas for these deter- 
minants. We use a simple involution to show that the Stanley hook- 
content formula (Theorem 7.21.2 of |Sta] ) follows from our formula. 

We connect these formulas to some special values of symmetric func- 
tions that are not obtained by evaluations. 





1 + aq'^^'^ 



1 — q^i'j 



where hij = Xi — i + 1 + X'j 
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The end of Section 3 is devoted to reverse plane overpartitions. A 
reverse plane (over) partition is defined like a plane (over)partition with 
nonincreasing property substituted with nondecreasing property and 
allowing O's as entries. Precisely, a reverse plane partition of shape A is 
a filling of cells of the Ferrers diagram of A with nonnegative integers 
that form a nondecreasing sequence along each row and each column. 
A reverse plane overpartition is a reverse plane partition where only 
positive entries can be overlined and in each row the last occurrence 
of an integer can be overlined or not and in each column the first 
occurrence of a positive integer can be overlined or not and all others 
(if positive) are overlined. An example of a reverse plane overpartition 
is 

3 4 4 4 

4 4 

1 3 
3 3 

It was proved by Gansner [G] that the generating function of reverse 
plane partitions of a given shape A is 



;i.5) n 



1 — (7 '-^ 
(ij)6A ^ 

Let S^{\) be the set of all reverse plane overpartitions of shape A. 
The generating function of reverse plane overpartitions is given by the 
following hook-formula. 

Theorem 5. Let X be a partition. The generating function of reverse 
plane overpartitions of shape A is 

ini = TT ^ + 

ne5«{A) (i,i)6A ^ 



We construct a bijection between reverse plane overpartitions of a 
given shape and sets of nonintersecting paths whose endpoints are not 
fixed. Using results of |Stej we obtain a Pfaffian formula for the gener- 
ating function of reverse plane partitions of a given shape. We further 
evaluate the Pfaffian and obtain a bijective proof of the hook formula. 
When A is the partition with r parts equal to c, this result is the gen- 
erating formula for plane overpartitions fitting in a box r x c, namely 
Theorem [H 

In Section 4 we make a connection between plane overpartitions and 
domino tilings. We give some basic properties of this correspondence, 
like how removing a box or changing a mark to unmark changes the 
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corresponding tiling. This, correspondence connects a measure on strict 
plane partitions studied in |V2j to a measure on domino tilings. This 
connection was expected by similarities in correlation kernels, limit 
shapes and some other features of these measures, but the connection 
was not established before. 

In Section 5 we propose a bijection between matrices and pairs of 
plane overpartitions based on Berele and Remmel [BeR] which gives a 
bijection between matrices and pairs of {k, /)-semistandard tableaux. 
This bijection is based on the jeu de taquin. We give another stronger 
version of the shifted MacMahon's formula, as we give a weighted gen- 
erating function of plane overpartitons with bounded entries. Let C{n) 
be the set of all plane overpartitions with the largest entry at most n. 

Theorem 6. The weighted generating functions of plane overpartitions 
where the largest entry is at most n is 



In Section 6 we study interlacing sequences and cylindric partitions. 
We say that a sequence of partitions A = (A-^, . . . , A'^) is interlacing 
if AYA*^^ or A*^^/A* is a horizontal strip, i.e. a skew shape having at 
most one cell in each column. Let A = {Ai, . . . , At-i) be a sequence of 
O's and I's. We say that an interlacing sequence A = (A^, . . . , A"^) has 
profile A if when Ai = 1, respectively Ai = then AYA*"*"^, respectively 
y+i^y is a horizontal strip. The diagram of an interlacing sequence is 
the set of boxes filled with parts of A*'s as shown in Figured Namely, 



the 2*'* diagonal represents A* and if the first part of A* is placed at {i,j) 
then the first part of A*"*"^ is placed at j — 1) if = 1 or (i+l, j + l) 
otherwise. Observe that the profile A defines the upper border of the 



ne£(n) 





Figure 2. The diagram of an interlacing sequence 
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diagram (part of the border with a positive slope corresponds to O's 
and with negative to I's). 

A (skew) plane partition and cylindric partition are examples of in- 
terlacing sequences. A plane partition can be written as A = (0, A^, . . . , 
with profile A = (0, 0, . . . , 0, 1, . . . , 1, 1) and A*s are diagonals of the 
plane partition. A skew plane partition is an interlacing sequence 
A = (0, A\ . . . , A^, 0) with a profile A = (Aq, Ai, . . . , At-i, At), where 
Aq = and A^ = 1. A cylindric partition is an interlacing sequence 
A = (A°, A^, . . . , A"^) where A° = A^, and T is called the period of A. A 
cylindric partitions can be represented by the cylindric diagram that 
is obtained from the ordinary diagram by identification of the first and 
last diagonal. 

A connected component of an interlacing sequence A is the set of all 
rookwise connected boxes of its diagram that are filled with a same 
number. We denote the number of connected components of A with 
k{A). For the example from Figure [2] we have k{A) = 19 and its con- 
nected components are shown in Figure [2] (bold lines represent bound- 
aries of these components). 

If a box {i,j) belongs to a connected component C then we define its 
level i{i,j) as the smallest positive integer such that (z + i,j + €) does 
not belong to C . In other words, a level represents the distance from 
the "rim" , distance being measured diagonally. A border component is 
a rookwise connected subset of a connected component where all boxes 
have the same level. We also say that this border component is of this 
level. All border components are rim hooks. For the example above, 
border components and their levels are shown in Figure O 




Figure 3. Border components with levels 

For a cylindric partition we define cylindric connected components 
and cylindric border components in the same way but connectedness 
is understood on the cylinder, i.e. boxes are connected if they are 
rookwise connected in the cylindric diagram. 
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Let (ni,n2, . . . ) be a sequence of nonnegative integers where rii is 
the number of z-level border components of A. We set 

(1.6) A^{t) = ii{i-tr'. 

i>l 

For the example above AA(t) = (1 - - - t^)- 

Similarly, for a cylindric partition 11 we define 

i>l 

where nf^ is the number of cylindric border components of level i. 

In Section 6 we give a generating formula for skew plane parti- 
tions. Let Skew(T, A) be the set of all skew plane partitions A = 
(0, A\ . . . , A'^, 0) with profile A = {Aq, Ai, . . . , At-i, At), where Aq = 
and At = 1. 

Theorem 7. ( Generalized MacMahon's formula for skew plane parti- 
tions; Hall-Littlewood's case) 

E ^nw,™= n 

UGSkew(T.A) 0<i<j<T ^ 

Ai=0,Aj=l 



Note that as profiles are words in {0, 1}, a profile A = {Aq, . . . , At) 
encodes the border of a Ferrers diagram A. Skew plane partitions of 
profile A are in one-to-one correspondence with reverse plane parti- 
tions of shape A. Moreover, one can check that 

l-r 1 - tq^-' _ -TT 1 - tq''''' 

11 I — (jj-i 11 I — (jhij ' 

0<i<j<T ^ (ij)eA ^ 

Ai=0,Aj=l 

Therefore the Theorem of Gansner (equation (11.51) ) is Theorem [7] with 
t = and our Theorem [5] on reverse plane overpartitions is Theorem [7] 
with t = —1. 

This theorem is also a generalization of results of Vuletic [V2] . In 
[V2j a 2-parameter generalization of MacMahon's formula related to 
Macdonald symmetric functions was given and the formula is especially 
simple in the Hall-Littlewood case. In the Hall-Littlewood case, this is 
a generating formula for plane partitions weighted by A]j{t). Theorem 
[7| can be naturally generalized to Macdonald case, but we don't pursue 
this here. 
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Let Cyl(T, A) be the set of all cylindric partitions with period T and 
profile A. The main result of Section 6 is: 

Theorem 8. (Generalized MacMahon's formula for cylindric parti- 
tions; Hall-Littlewood's case) 

^uitW^^ = n i-a^T n 1 _ Jj~i)m+in-i)T ' 

neCyl(T,A) n=l ^ l<i,j<T ^ 

Ai=0,Aj=l 

where i(T) is the smallest positive integer such that i = i(T) mod T . 

The case t = is due to Borodin and represents a generating for- 
mula for cylindric partitions. Cylindric partitions were introduced and 
enumerated by Gessel and Krattenthaler [GK] . The result of Borodin 
could be also proven using Theorem 5 of |GK] and the SU(r) -extension 
of Bailey's qiIjq summation due to Guftavson (equation (7.9) in [GK] ) 
|Kr] . Again Theorem[H]can be naturally generalized to Macdonald case. 
The trace generating function of those cylindric partitions could also 
be easily derived from our proof, as done by Okada [O] for the reverse 
plane partitions case. 

The paper is organized as follows. In Section 2 we give a mostly 
combinatorial proof of the generalized MacMahon formula. In Section 
3 we use nonintersecting paths and Gessel- Viennot result's to obtain 
the hook-length formulas for plane overpartitions for reverse plane par- 
titions of a given shape. In Section 4 we make the connection between 
tilings and plane overpartitions. In Section 5 we construct a bijection 
between matrices and pairs of plane overpartitions and obtain a gener- 
ating formula for plane overpartitions with bounded part size. In Sec- 
tion 6 we give the hook formula for reverse plane partitions contained 
in a given shape and the 1-parameter generalization of the generating 
formula for cylindric partitions. 

Acknowledgment. The authors want to thank Alexei Borodin, the 
advisor of one of the authors, for help and guidance and Dominique 
Gouyou-Beauchamps, Richard Kenyon and Cedric Boutillier for useful 
discussions. 



2. Plane partitions and Hall-Littlewood functions 

In this Section, we give an alternative proof of the generalization of 
MacMahon formula due to the third author [V2] . Our proof is mostly 
combinatorial as it uses a bijection between plane partitions and pairs 
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of strict plane partitions of the same shape and the combinatorial de- 
scription of the Hall-Littlewood functions [Macj . 

Let P(r, c) be the set of plane partitions with at most r rows and 
c columns. Given a plane partition 11, let Aii{t) be the polynomial 

defined in m. as An(t) = Ur border component(l ' ^'^™'^^^)- 

Recall that Theorem [2] states that 



ne-p(r,c) i=l j=l 



Any plane partition 11 is in bijection with a sequence of partitions 
{7T^^\ 7r^^\ . . .). This sequence is such that vr*^*) is the shape of the entries 
greater than or equal to z in 11. 

4433 

For example if 11 = 3332 , the corresponding sequence is ((4, 4, 1), (4, 4), (4, 3), (2)). 
1 

Note that the plane partition A is column strict if and only if A'-^^/A^*"*"^-* 
is a horizontal strip. 

We use a bijection between pairs of column strict plane partitions 
(S,A) and plane partitions 11 due to Bender and Knuth |BK] . We 
suppose that (S, A) are of the same shape A and that the corresponding 
sequences are {a^^\ a^'^\ . . .) and (A'-^-*, A*^^-*, . . .) 

Given a plane partition 11 = (Hj j), we define the entries of diagonal 
X to be the partition (Hij) with i,j>l and j — i = x. The bijection is 
such that the entries of diagonal a; of 11 are a^^^^-* if a; > and A'-"^"^^ 
otherwise. Note that as A and E have the same shape, the entries of 
the main diagonal (x = 0) are a'-^-' = \^^\ 

4444 

For example, start with E = 2221 whose sequence is ((4, 4, 3), (4, 3), (4), (4)) ; 

111 

4433 

and A = 3322 , whose sequence is ((4, 4, 3), (4, 4), (4, 2), (2)) and get 
111 

4444 
443 
443 ■ 
22 
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This construction implies that : 

|n| = |S| + |A|-|A|; 



with 



and 



hit) 



i>i j=i 



i>l 



given a horizontal strip 6 = X/fi, 

where mj(A) is the multiplicity of i in A and / is the set of integers such 
that d'i = 1 and O'^^^ = 0. See [Mac] . Chapter III Sections 2 and 5. 
Indeed 

• Each factor (1 — f) in b\(t) is in one-to-one correspondence 
with a border component of level i that goes through the main 
diagonal of 11. 

• Each factor (1 — t*) in is in one-to-one correspondence 
with a border component of level i that ends in a non-negative 
diagonal. 

• Each factor (1 — f) in 0a (^) is in one-to-one correspondence 
with a border component of level i that starts in a non-positive 
diagonal. 

Continuing with our example, we have 

Mt) = {i-tni-n Mt) = {i-tf{i-n b,{t) = (i-t)2(i-t2). 

and 

A„it) = (1 - 0^(1 - a = (,.tniJ) — - 

We recall the combinatorial definition of the Hall-Littlewood func- 
tions as done in the book of Macdonald [Mac] . The Hall-Littlewood 
function Qx{x;t) can be defined as 



Qx{x;t) = J2 



A 

sh(A)=A 
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where and ttj is the number of entries equal to i in A. 

See |Macj Chapter 111, equation (5.11). 

A direct consequence of the preceding bijection is that, the entries 
of S are less or equal to r and the entries of A are less or equal to c if 
and only if 11 is in ^(r, c). Therefore : 

A mini QA(g, •••,g^o, ...;t)QA(g°,---,g'^\o, ...;t) 
^ ^n(t)g ^ . 

Finally, we need equation (4.4) in Chapter 111 of [Mac] . 

With the substitutions Xj = for 1 < i < r and otherwise and 
yj = q^~^ for 1 < j < c and otherwise, we get the result. 

3. NONINTERSECTING PATHS 

3.1. Plane overpartitions of a given shape. In this section we rep- 
resent plane overpartitions as nonintersecting paths and using results 
from [GVj we prove Theorems [3] and |H A similar approach was used 
for example in |Brj to compute Super-Schur functions. 

We construct a bijection between the set of paths from (0, 0) to (x, k) 
and the set of overpartitions with at most k parts less or equal to x. 
Given an overpartition the corresponding path consists of North and 
East edges that form the border of the Ferrer diagram of the over- 
partition except for corners containing an overlined entry where we 
substitute a pair of North and East edge with an North-East edge. 
For example, the path corresponding to the overpartition (5, 5, 3, 3, 2) 
is shown on Figure |H 



Figure 4. Paths and overpartitions 

To each overpartition A we associate a weight equal to a"^^\^^^^ where 
o(A) is the number of overlined parts. To have the same weight on the 
corresponding path we introduce the following weights on edges. We 
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assign weight 1 to East edges, g* to North edges on level i and weight 
ag*+^ to North-East edges joining levels i and ^ + 1. For convenience 
we use these paths and weights to construct a bijection between plane 
overpartitions and sets of nonintersecting paths. 

Lemma 1. |CLj The generating functions of overpartitions with at 
most k parts is given by 



(3.1) 



l{X)<k 

and exactly k parts is given by 



(3.2) 



J2 

l(X)=k 



Proof. Let A be an overpartition seen as the corresponding path. Let 
rj(A) be the number of North-East edges joining levels i and i + 1. 
Obviously, it takes values or 1. Let Sj(A) be the number of East 
edges on level i. Then 



(A)^|A| 



l(\)<k 



k k oo 

nEwiiEM' 

i=l ri=0,l 



-aq)k 



Now, if A is an overpartition with exactly k parts then r/(.(A) or Sfc(A) 
must be nonzero. Then using (13.11) we obtain 



l{X)=k 



-aq)k 



+ a- 



-aq) 



k-l 



-ajk 



□ 



For a plane overpartition 11 of shape A we construct a set of nonin- 
tersecting paths using paths from row overpartitions where the starting 
point of the path corresponding to the ith row is shifted upwards by 
Ai — Aj + i — 1 so that the starting point is (0, Ai — Aj + z — 1). That way 
we obtain a bijection between the set of nonintersecting paths from 
(0, Ai — Aj + ? — 1) to {x, \i + i — 1), where i goes from 1 to and the 
set of plane overpartitions whose ith row overpartition has at most Aj 
and at least Aj_i parts with x greater or equal to the largest part. The 
weights on the edges correspond to the weight a°^^^q^^^. Figure [5] (see 
also Figure [8]) shows the corresponding set of nonintersecting paths for 
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X = 8 and the plane overpartition 

7 4 

3 3 

3 2 
2 



Figure 5. Nonintersecting paths 

Definition 1. For a partition A we define Mx{a; q) to be a i{X) x £{X) 
matrix whose {i,j)th entry is given by 

{-a)x^+i-j 

For a partition A let B{X) and S{\) be as in the introduction, i.e. the 
sets of all plane overpartitions bounded by shape A and of the shape 
A, respectively. 

Proposition 1. Let X be a partition. The weighted generating function 
of plane overpartitions such that the i^^ row of the plane overpartition 
is an overpartition that has at most Aj parts and at least Aj+i parts is 
given by 

J2 a°(")gl°' = detMA(ag;g). 

neB(A) 

Proof. From (13.1 p we have that the limit when x goes to infinity of the 
number of paths from (0, 0) to (x, k) is {—aq)k/{q)k- Using Corollary 2 
of |GV] we have that det M\{aq] q) is the limit when x goes to infinity 
of the generating function of i{X) nonintersecting paths going from 
(0, Xi+i — l—Xi) to {x, Xi+i—1). Thanks to the bijection between paths 
and overpartitions this is also the generating function for overpartitions 
whose zth row overpartition has at most A^ and at least Aj+i parts. □ 



3 2 2 
3 2 
2 
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Proposition 2. Let \ be a partition. The weighted generating function 
of plane overpartitions of shape X is given by 

nG5{A) 

Proof. The proof is the same as the proof of Proposition [H Here we 
just use (13.21) instead of (13.11) . □ 

We now prove a lemma that will allow us to give a product formula 
for these generating functions. 

Lemma 2. 

(3.3) detM.(a;g)=g^»(-i)^' H T^^^" 

Proof. We prove the lemma by induction on the number of columns of 
A. For the empty partition the lemma is true. So we assume that A 
is nonempty and that for all fi having less columns then A the lemma 
holds, i.e. 

(3.4) detM,{a-q) = q^'^'-'^'^' J] \ ^ " " 

(«j)gm 

Suppose that £{\) = k. Let M = Mx{a; q) and Q = Mj{aq; q), where 
A = (Ai — 1, . . . , Afc — 1). Therefore, 

Our goal is to show that 

(3.5) det M = g(^) J] ^-^^ det Q, 

i=i ^ 

because then (13. 3p follows by induction. 

We introduce two k x k matrices and P with the following prop- 
erties: 

detM -A- {-a)h,^, 

(3.6) ^ ^Ui^kT' 

(3.7) d„/p:= ,nt^(a + .-),^(;)nL(l 



detP (l + a)'= (1 + a) 



^^/«^= d^ =|Jr^ 
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These properties imply f l3.5p . 
Matrices N and P are given by 

^^^^^^ = Mi^,J)^ = ;7^^-"'7^^|^-- 

and 

^-^ = ^-^ (g)fe.M-i ^ (-ag)A,+^-i-i(g)h,,i-i 
(-ag)/.,,i-i (g)A,+i-j-i(-ag)h,,i-i' 
Properties fl3.6p and (13.81) follow immediately. We will prove that 

(3.9) P(,,)^ (lH-.)iV(.,,)-(l-.--)P(. + U) _ 

a + 

which will imply fl3.7p . 
Let 

S={1 + a)N{i,j) - (1 - g^-OP(z + 
Note that /ij i = Xj — j + k. Then 

(1 + a)(-a)A,+»-j(g)h,. 1 (1 - g^~0(-«9)A,+i-i(g)h,,i-i 



(?)A,+i-j(-a)/»,,i (g)A,+i_j(-ag)h,,i 
(1 + a)(l - g^^-i) (1 - q''-'){l + ag^i+*-^) 



= {a + q'-')P{t,j) 
and thus (13.91) holds. 



□ 



Then using this lemma we obtain the product formulas for the gener- 
ating functions from Propositions [T] and [2J Those are the hook-length 
formulas given in Theorems [3] and HI 

Stanley's hook-content formula states the generating function of col- 
umn strict plane partitions of shape A where the entries are less or 
equal to n is 



n - 

(ij)6A ^ 



See Theorem 7.21.2 of [Staj . It can be seen as a special case of the 
hook-content formula from Theorem [3] for a = — g". Indeed there is an 
easy sign reversing involution on plane overpartitions where the over- 
lined entries are larger than n and where the sign of each entry is - if 
the entry is overlined (and greater than n) and -|- otherwise and the 
sign of the overpartition is the product of the signs of the entries. We 
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define now this involution. If tlie largest entry of the plane overparti- 
tion is greater than n then if it appears overlined in the first row then 
take off the over line and otherwise overline the rightmost occurrence of 
the largest entry in the first row. Note that this changes the sign of the 
plane overpartition. If the largest entry is at most n then do nothing. 

Now, we want to obtain a formula for plane overpartitions with at 
most r rows and c columns 

Proposition 3. The weighted generating function of plane overparti- 
tions with at most r rows and c columns 

^ det M(c,Ai,Ai,...,A(,_i)/2,A(,_i)/2)(a; q) 

c>Ai>...>A(r_i)/2>0 

if r is odd and 

detM(A,,Ai,...A/2,A./2)(«;?) 

n>Ai>...>Ar/2>0 

otherwise. 

In particular, the weighted generating function of all overpartitions 

is: 

detM(A,,Ai,...,Afc,A,)(a;g) 

Ai>...>Afe 

Proof. This is a direct consequence of Proposition [H □ 

We will use this result to get another "symmetric function" proof 
the shifted MacMahon formula. 

3.2. The shifted MacMahon formula. The shifted MacMahon for- 
mula obtained in [FWl EH |V2]: 

n is a plane n=l ^ ' 

overpartition 

can be obtained from symmetric functions using the results of the pre- 
ceeding Subsection. 

We showed how Stanley's hook-content formula follows from Theo- 
rem [31 Alternatively, we could use Stanley's hook-content formula and 
symmetric functions to prove Lemma [2] that further implies Theorems 
laandU 

A specialization is an algebra homomorphism between the algebra 
of symmetric functions and C. We use three standard bases for the 
algebra of symmetric functions: Schur functions s, complete symmetric 
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functions h and power sums p (see |Mac] ). If p is a specialization we 
denote their images with s|p, h\p and p\p. 

Let p{a) : A C[a] be an algebra homomorphism given with 

, I _ (-a)n 

Hn\,ia) - ^^^^ . 

Since, for A = (Ai, A2, • . • , A^) 

sx = det{hxi-i+j) 
we have from Definition [T] that 
(3.10) 

■^A |p(a) 

Now, we show that 



Pn\p{a) - ^_ ■ 

This follows from (2.2.1) of [X]: 



irr (9)„ 1 - tr 

and 

. ^ H'(t) 

where H and P are generating functions for /i„ and p„. 

Recall the well known formula (see for example Theorem 7.21.2 of 
[Stil): 

(3.11) ..(1, g, . . . , g", 0, . . . ) = J] • 

Observe that p(g") is a specialization given with the evaluation Xi = 
for i = 1, . . . ,n and otherwise. Then (13.111) says 

I — qnqc{x)+i 



^ - q" 



This implies that 



1 + ag 



c{x)+l 



because we have polynomials in a on both sides and equality is satisfied 
for infinitely many values (a = g", for every n). So, this gives an 
alternate proof of Lemma [2l 
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The shifted MacMahon formula can be obtained from (l3.1Up and 
Proposition [31 We have that 

= ^ S(Ai,Ai,A2,A2,...,Afc,Afe)lpi = -^^1 
Ai>A2>--->Ai. A' even 



I Pi) 



n is a plane 
ovcrpartition 



where A' is the transpose of A and a partition is even if it has even 
parts and pi = p(l). 

By Ex. 5 b) on p. 77 of [Macj we have that 



A' even 



i<j 



exp 



oo ^ 
n=l 



So, 



A' even 



exp 



exp 



exp 



^ 2n ( ( 1 



.71=1,3,5,, 



1 + q''- 



."=1,3,5,, 



nil - 



P2 



.n=l,3,5 

where the speciahzation p2 is given with 

^"Ip2 = i_ 

This means that the speciahzation p2 is actually an evaluation given 
with Xi = g(2j+i)/2_ gince 



exp 



n=l,3.5,. 



n 



Hi 



1 + XiX 



we obtain 



exp 



.n=l,3,5,. 



n 



-pI 



P2 



n=l 



which proves the shifted MacMahon formula. 

3.3. Reverse plane overpartitons. In this Section we construct a 
bijection between the set of all reverse plane overpartitions and sets 
of nonintersecting paths whose endpoints are not fixed. We use this 
bijection and Stembridge's results [Stej to obtain a Pfaffian formula for 
the generating function of reverse plane overpartitions of a given shape. 
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Evaluating the Pfaffian we obtain the hook formula for reverse plane 
overpartitions due to Okada. Namely, if S^{X) is the set of all reverse 
plane partitions of shape A then 

Theorem 9. The generating function of reverse plane overpartitions 
of shape X is 



ne5«(A) 



nf 



We construct a weight preserving bijection between reverse plane 
overpartitions and sets of nonintersecting paths on a triangular lat- 
tice in a similar fashion as in Section 2. The lattice consists of East, 
North and North-East edges. East edges have weight 1, North edges 
on level i have weight and North-East edges joining levels i and i + 1 
have weight g*. The weight of a set of nonintersecting paths p is the 
product of the weights of their edges and is denoted with w{p). Let 
n be a reverse plane overpartition whose positive entries form a skew 
shape X/fi and let i = i{X). Then 11 can be represented by a set of n 
nonintersecting lattice paths such that 

• the departure points are (0, fii + i — i) and 

• the arrivals points are (x, Xi + i — i), 

for a large enough x and i = For example let x = 8, A = 

(5,4,2,2) and /i = (2,1). Figure [6] shows the corresponding set of 
nonintersecting paths for the reverse plane overpartition of shape X\fi 

3 4 4 
3 4 4 
1 3 
3 3 




Figure 6. Nonintersecting paths and reverse plane overpartitions 

This implies that all reverse plane overpartitions of shape A can be 
represented by nonintersecting lattice paths such that 
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• the departure points are a /-element subset of {(0,z) \i > 0} 
and 

• the arrivals points are (x, Aj + / — i), 
with X oo. 

Now, for ri > r2 > ■ ■ ■ > r„ > we define 

iy(ri,r2, . . . ,r„) = lim w{p), 

x— ►oo ' ' 

peP{x;ri,...,rn) 

where P{x] Vi, r2, . . . , r^) is the set of all nonintersecting paths joining 
a n-element subset of {{0,i)\i > 0} with {(x, ri), . . . , (x, r„)}. Note 
that for ri > r2 > ■ ■ ■ > r„ > 

(3.12) Win, . . . , r„, 0) = Win - 1, . . . , - 1). 

By Stembridge's Pfaffian formula for the sum of the weights of nonin- 
tersecting paths where departure points are not fixed [Stej we obtain 

iy(ri,r2,...,r„) = Pf(D), 

where if n is even D is nxn skew-symmetric matrix defined by Dij = 
Wivi, Tj) for 1 < i < j < n and if n is odd D is (ra + 1) x (n + 1) skew- 
symmetric matrix defined by Di j = Wiri,rj) and -Dj,„+i = W^(rj) for 
l<i<j<n + l. 

Lemma 3. Let r > s > 0. Then 



[-<i)r i-q). 



^r—s 



(3.13) Wis) = 

(3.14) Wir,s) = 

iq)r iq)s 1 + 

Proof. From Lemma [T] we have that the generating function of overpar- 
titions with at most n parts is M(n) = i—q)n/iq)n and with exactly n 
parts is Pin) = q'"(— l)n/(Q')n- We set Min) = Pin) = 0, for negative 
n. 

From this lemma it follows: 

n 

(3.15) ^P(z)=M(n). 

1=0 

Thus, 

Wis) = Mis) = and Wir, 0) = M(r - 1) = t^tzi. 

iq)s (g)r-i 

We prove (13.141) by induction on s. The formula for the base case 
s = holds by above. So, we assume s > 1. 
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By Gessel-Viennot's determinantal formula |GV] we have that 

s r 

W{r, s) = J2 J2 ^ " ^) " ~ " 

i=0 j=i+l 

Summing over j and using (13.151) we obtain 

s 

W{r, s) = Y^ P{s - i)M{r - i - 1) - P(r - i)M{s -l-i). 

1=0 

Then 

W{r, s) = W{r - 1, s - 1) + P{s)M{r - 1) - P{r)M{s - 1). 
It is enough to prove that 

P{s)M{r - 1) - P{r)M{s - 1) _ 2{q^ + q^) 
^ ' W{r-l,s-l) {l-q-){l-q^) 

and fl3.14p follows by induction. Now, 

P{s)M{r - 1) - P{r)M{s - 1) = 

(g), (g)^_i (g),. 
(-g)r-i (-g).-i 1 + 2(g^ + g^) 



Using inductive hypothesis for W{r — 1, s — 1) we obtain (13.161) . 



□ 



Let F\ be the generating function of reverse plane overpartitions of 
shape A. Then using the bijection we have constructed we obtain that 

Fa = W^(Ai + /-1,A2 + /-2,...,A0 

which, after applying Stembridge's result, gives us the Pfaffian formula. 
This Pfaffian formula can be expressed as a product after the following 
observations. 

Let M be 2A; X 2k skew-symmetric matrix. One of definitions of the 
Pfaffian is the following: 

Pf(M) = sgn(n)M,„,,M,„,, ■ ■ ■ M,„,,„, 

n=(n,Jl)...(in Jn) 
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where the sum is over all of perfect matchings (or fixed point free 
involutions) of \^n\. Also, for r > s > 

W{s) = \±^W{s-l) 
W(r,s) = -—^■-—^W(r-l,s-l) 



Then 

Fa = n j I ■ ^(^1 + / - 2, A2 + / - 3, . . . , Az - 1) = F; 



I 



A) 



where A = (Ai — 1, . . . , A; — 1) if A; > 1 and A = (Ai — 1, ... , A;_i — 1) 
if A; = 1 (see fl3.12p in this case). Inductively we obtain Theorem [91 

4. Domino tilings 

In |Vlj a measure on strict plane partitions was studied. Strict plane 
partitions are plane partitions were all diagonals are strict partitions, 
i.e. strictly decreasing sequences. They can also be seen as plane 
overpartitions where all overlines are deleted. There are 

2Mn) different 

plane overpartitions corresponding to a same strict plane partition 11. 

Alternatively, a strict plane partition can be seen as a subset of N x Z 
consisting of points (t, x) where a; is a part of the diagonal partition in- 
dexed by t, see Figure [71 We call this set the 2-dimensional diagram of 
that strict plane partition. The connected components are connected 
sets (no holes) on the same horizontal line. The 2-dimensional diagram 
of a plane overpartition is the 2-dimensional diagram of its correspond- 
ing strict plane partition. 




Figure 7. A 2-dimensional diagram 

The measure studied in jVlj assigns to each strict plane partition 
a weight equal to 2''^^^q^^^. The limit shape of this measure is given 
in terms of the Ronkin function of the polynomial P{z,w) = — 1 + 
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z + w + zw and it is parameterized on the domain representing half 
of the amoeba of this polynomiaL This polynomial is also related 
to plane tilings with dominoes. This, as well as some other features 
like similarities in correlation kernels, see [Jl IVl] , suggested that the 
connection between this measure and domino tilings is likely to exist. 

Alternatively, one can see this measure as a uniform measure on 
plane overpartitions, i.e. each plane overpartition 11 has a probabil- 
ity proportional to gl'^L In Section [3] we have constructed a bijection 
between overpartitions and nonintersecting paths passing through the 
elements of the corresponding 2-dimensional diagrams, see Figure [HI 
The paths consist of edges of three different kinds: horizontal (joining 
(t, x) and (t + 1, x)), vertical (joining {t, a; + 1) and {t, x)) and diagonal 
(joining {t,x + l) and {t + l,x)). There is a standard way to construct a 
tiling with dominoes using these paths, see for example [J]. We explain 
the process below. 




Figure 8. A strict plane partition and the domino tiling 

We start from and color it in a chessboard fashion such that 
(1/4, 1/4), (-1/4, 3/4), (1/4, 5/4) and (3/4, 3/4) are vertices of a white 
square. So, the axes of this infinite chessboard form angles of 45 and 135 
degrees with axes of R^. A domino placed on this infinite chessboard 
can be one of the four types: (1, 1), (—1, —1), (—1, 1) or (1, —1), where 
we say that a domino is of type {x, y) if (x, y) is a vector parallel to 
the vector whose starting, respectively end point is the middle of the 
white, respectively black square of that domino. 

Now, take a plane overpartition and represent it by a set of non- 
intersecting paths passing through the elements of the 2-dimensional 
diagram of this plane overpartition. We cover each edge by a domino 
that satisfies that the endpoints of that edge are middle points of sides 
of the black and white square of that domino. That way we obtain 
a tiling of a part of the plane with dominoes of three types: (1,1), 
(—1,-1) and (1,-1). More precisely, horizontal edges correspond to 
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(1,1) dominoes, vertical to (—1,-1) and diagonal to (1,-1). To tile 
the whole plane we fill the rest of it by dominoes of the forth, (—1,1) 
type. See Figure [8] for an illustration. 

This way we have established a correspondence between plane over- 
partitions and plane tilings with dominoes. We now give some of 
the properties of this correspondence. First, we describe how a tiling 
changes in the case when we add or remove an overline or we add or 
remove a box from a plane overpartition. We require that when we 
add/remove an overline or a box we obtain a plane overpartition again. 
It is enough to consider only one operation adding or removing since 
they are inverse of each other. In terms of the 2-dimensional diagram of 
a plane overpartition, adding an overline can occur at all places where 
[t, x) is in the diagram and {t + 1, x) is not. Removing a box can occur 
at all places where {t, x) is the diagram and {t, a; — 1) is not. 

Adding an overline means that a pair of horizontal and vertical edges 
is replaced by one diagonal edge. This means that the new tiling differs 
from the old one by replacing a pair of (1,1) and (—1,-1) dominoes 
by a pair of (1,-1) and (—1, 1) dominoes, see Figure O 



Figure 9. Adding an overline 

Removing a box from a plane overpartition corresponds to a similar 
thing. Observe that if a box can be removed then the corresponding 
part is overlined or it can be overlined to obtain a plane overpartition 
again. So, it is enough to consider how the tiling changes when we 
remove a box from an overlined part since we have already considered 
the case of adding or removing an overline. If we remove a box from 
an ovelined part we change a diagonal edge by a pair of vertical and 
horizontal edges. This means that the new tiling differs from the old 
one by replacing a pair of (1,-1) and (—1, 1) dominoes by a pair of 
(1,1) and (—1,-1) dominoes, see Figure [TOl The difference between 
removing an overline and removing a box is just in the position where 
the flipping of dominoes occurs. 

We conclude this section by an observation that plane overpartitions 
of a given shape A and whose parts are bounded by n are in bijection 
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Figure 10. Removing a box 

with domino tilings of a rectangle [— ^(A) + 1, Ai] x [0,n] with certain 
boundary conditions. These conditions are imposed by the fact that 
outside of this rectangle nonintersecting paths are just straight lines. 
We describe the boundary conditions precisely in the proposition below. 

Proposition 4. The set S{X) U C{n), of all plane overpartitions of 
shape A and whose largest part is at most n, is in a bijection with plane 
tilings with dominoes where a point (t,x) G Z x R is covered by a 
domino of type (—1, —1) if 

• t < -£(A), 

• — ^(A) < t < and x > n, 

• t = Xi — i + 1 for some i and x < 0, 

and a domino of type (—1,1) if 

• t > Ai, 

• < t < Ai and x > n + 1/2, 

• t ^ Xi — i + 1 for all i and x < —1/2. 

For the example from Figure [H] the boundary conditions are shown 
in Figure [TTl 




Figure 11. Boundary conditions 
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5. Robinson-Schensted-Knuth (RSK) type algorithm for 

plane overpartitions 

In this section we are going to give a bijection between matrices 
and pairs of plane overpartitions of the same shape. This bijection is 
inspired by the algorithm RS2 of Berele and Remmel |BeRj which gives 
a bijection between matrices and pairs of {k, £)-semistandard tableaux. 
It is shown in |Savj that there is a bijection between [n, r2)-semistandard 
tableaux and plane overpartitions whose largest entry is at most n. 
This bijection is based on the jeu de taquin. 

We then apply properties of this algorithm to enumerate plane over- 
partitions, as done by Bender and Knuth [BKj for plane partitions. 

5.1. The RSK algorithm. We start by describing a bijection between 

matrices and pairs of numbers. Let M = be a 2n x 2n matrix, 

made of four n x n blocks A,B,C and D. The blocks A and D are 
non-negative integer matrices, and B and C are {0, 1} matrices. The 
encoding of M into pairs is made using the following rules: 

• for each non-zero entry of A, we create ajj pairs 

• for each non-zero entry bij of B, we create one pair 

• for each non-zero entry of C, we create one pair 

• for each non-zero entry dij of we create dij pairs 







It is clear that this encoding defines a one-to-one correspondence 
between matrices and pairs of numbers. 



Example 1. Let M 



/O 2 1 0\ 
2 10 
110 1 

yo 1 ly 



. After encoding M, we obtain 



From now, we fix the order 1<1<2<2<3<3<..., we sort the 
pairs to create a two-line array L such that : 
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• the first line is a non-increasing sequence 

• if two entries of the first line are equal and overlined (resp. non- 
overlined) then the corresponding entries in the second line are 
in weakly increasing (resp. decreasing) order. 

Example 2 (example 1 continued). After sorting, we obtain the two- 
line array L 

/2,2,2,2,2,1,1,1,1,1,1\ 
1^1, 1,1, 1,2, 2, 2, 1,1, 2, 2^1 

We now describe the part of the bijection which is the insertion al- 
gorithm. It is based on an algorithm proposed by Knuth in [K] and 
quite similar to the algorithm RS2 of [BeRj . 

We first define how to insert the entry j into a plane overpartition 

P. 

(2) Xi ^ J 

(3) If Xi is smaller than all entries of the z*'^ row of P, then insert 
Xi at the end of this row. 

(4) Otherwise let ji the index such that Pi,ji~i > Xi > Piji- 

(a) If -Pjji-i = Xi and Pi^ij-i is overlined, then Xj+i <— Xi. 

(b) Otherwise 

(i) Xj+i ^ Pjj; 

(ii) 

(5) i i -\- 1. Go to step 3. 



Now we define how to insert a pair 




into a pair of plane over- 



partitions of the same shape (P, Q). 

(1) Insert j in P. 

(2) If the insertion ends in column c and row r of P, then insert i 
in column c and row r in Q. 

Now to go from the two-line array L to pairs of plane overpartitions 
of the same shape goes as follows : start with two empty plane over- 
partitions and insert each pair of L going from left to right. This is 
identitical to the classical RSK algorithm [K]. 
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Continuing the previous example, we get 

2 2 2 2 2 2 

2 11 2 11 

1 > ^ 1 

1 1 
1 1 

Theorem 10. There is a one-to-one correspondance between matrices 
M and pairs of plane overpartitions of the same shape {P,Q). In this 
correspondance : 

• k appears in P exactly J2i ^ik + Qfc 

• k appears in P exactly ^ • hik + dik 

• k appears in Q exactly + hik 

• k appears in Q exactly Cik + dik 

Proof. The proof is really identical the proof of the RSK algorithm [K] 
or the RS2 algorithm |BeRj . Details are given in [Sav] . □ 

As in [K], we can show that : 

Theorem 11. If insertion algorithm produces {P, Q) with input matrix 
M, then insertion algorithm produces {Q,P) with input matrix M*. 

Proof. The proof is again analogous to [K]. Given a two-line array 

Ui, . . . ,un \ we partition the pairs {ui.vA such that (uk,Vk) and 
Vi, . . . ,Vn J 

{Um, Vrn) with are in the same class if and only if : 

• Uk < Um and if Uk = Um, then Uk is overlined AND 

• Vk > Vm and if Vk = Vm, then Vk is overlined. 

Then one can sort each class so that the first entry of each pair appear 
in non-increasing order and then sort the classes so that the first entries 
of the first pair of each class are in non-increasing order. For example 
if the two-line array is 

'2,2,2,2,2,1,1,1,1,1,1 
1,1,1,1,2,2,2,1,1,2,2 

we get the classes 

Ci = {(2,1), (2,1), (1,2), (1,2)}, C2 = {(1,1)}, C3 = {(1,1), (1,1), (1,1)} 
If the classes are Ci, . . . ,Cd with 
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then the first row of P is 



and the first row of Q is 



Udd- 



Moreover one constructs the rest of P and Q using the pairs : 



See Lemma 1 of [K] and [Sav] for a complete proof. As the two-line 
array corresponding to is obtained by interchanging the two lines 
of the array and rearranging the columns, the Theorem follows. □ 

This implies that M = if and only if P = Q. Therefore : 

Theorem 12. There is a one-to-one correspondance between symmet- 
ric matrices M and plane overpartitions P. In this correspondance : 

• k appears in P exactly ^ • + Cj^ 

• k appears in P exactly ^ ,• hik + dik 

5.2. Enumeration of plane overpartitions. From the Theorem [T2l 
we can get directly the generating function for plane overpartitions 
whose largest entry is at most n. These objects are in bijection with 
symmetric matrix M of size 2n x 2n with blocks A, B, C and D, each 
of size nxn. The weight of the plane overpartition is ^ • j i{aij + bi j + 
Cij-\-dijy As M is symmetric, the weight of the plane overpartition can 
also be written as Y,i,j + Si + + Y.j<i +i) (oj j + c^ij) • 
Let On{m, k) be the number of plane overpartitions of m with k over- 
lined parts and entries at most n and 0„(g,a) = J2m k^ni'nT', k)q"^a^ . 



Theorem 13. The generating function On{q,a) for plane overparti- 
tions whose largest entry is at most n is 



Let 0{q, a) be the limit of C„(g, a) when n goes to infinity. A direct 
consequence of the Theorem is the following. 

Corollary 1. The generating function for plane overpartitions 0{q,a) 
is 




Then 



oo 



(1 + aq^y- 



n 



32 SYLVIE CORTEEL', CYRILLE SAVELIEF' AND MIRJANA VULETIC* 

We can also get some more general result, as in |BKj : 

Theorem 14. The generating function of plane overpartitions whose 
parts lie in a set S of positive integers is : 

( \ 



(5.1) n 



iG5 



;i - (t){\ - act) n (1 - - ag«+^) 



For example 

Corollary 2. The generating function for plane overpartitions into odd 
parts is 

(1 + aq^y-^ 



J. (1 _ q2i-i){i _ ag2^-i)(l _ g2^)L^/2J(l - ag20LV2j " 
6. Interlacing sequences and cylindric partitions 



We want to combine results of |Boj and |V2j to obtain a 1-parameter 
generalization of the formula for the generating function of cylindric 
partitions related to the Hall-Littlewood symmetric functions. 

We use definitions of interlacing sequences, profiles, cylindric parti- 
tions, polynomials Anif) and A'^it) given in Introduction. 

For an ordinary partition A we define a polynomial 

(6.1) &A(t)=n^-»(A)(^)' 

i>l 

where mj(A) denotes the number of times i occurs as a part of A and 

ifr{t) = (i-t)(i-t2)...(i-r). 

For a horizontal strip 6 = X/fi we define 

/A/M = {^>l|^.' = land ei^,=0} 
JA/M = {j>l|^- = Oand^;.+i = l} 

Let 

(6.2) ^,/,it) = n (1 - ^'"'^'^) ^vm(^) = n (1 - ^'"^^'^)- 

Then 

V'a/mM/m = hx{t)/h^{t). 
For an interlacing sequence A = (A^ . . . , A^) with profile A = (Ai, . . . , At-i) 
we define $A(t): 

T-l 

(6.3) $^(t) = J]0,(t), 



1=1 
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where 



For A = (A\ . . . , A"^) and M = . . . , fi^) such that = fj,^ we 



A-M=(A\...,A^ 



, fi , . . . , 

An M = A 



define 
and 
Then 

(6.4) Aa.m = and <I>a.a/ = $a$a/. 

For an interlacing sequence A = (A^, . . . , A^) with profile A = {Ai, . . . , At-i] 
we define the reverse A = (A"^, . . . , A^) whose profile is A = (1 — 
At-u-.-A- A)- Then 

(6.5) ^a = Aa and $^ = 

For an ordinary partition A we construct an interlacing sequence 
(A) = (0, \\..., A^) of length L + 1 = /(A) + 1, where A* is obtained 
from A by truncating the last L — i parts. Then 

(6.6) A^x) = <f (A) = ^A- 

In our earlier paper [V2] (Propositions 2.4 and 2.6) we have shown 
that for a plane partition 11 

(6.7) $n = ^n. 
Now, more generally 

Proposition 5. If A = {X^, . . . , A"^) is an interlacing sequence then 

Oai 

Proof. If we show that the statement is true for sequences with con- 
stant profiles then inductively using (16. 4p we can show it is true for 
sequences with arbitrary profile. It is enough to show that the state- 
ment is true for sequences with (0, . . . , 0) profile because of (16. 5p . 
So, let A = (A^, . . . , A^) be a sequence with (0, . . . , 0) profile. Then 
n = (A^) ■ A ■ (A^) is a plane partition and from (16.41) . (16. 5p . (16. 6p and 
(jO) we obtain that An = A^ and $n = ^ai^a- □ 

For skew plane partitions and cylindric partitions we obtain the fol- 
lowing two corollaries. 
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Corollary 3. For a skew plane partition 11 we have that $n = ^n- 

Corollary 4. If U is a cylindric partition given with A = (A°, . . . , A"^) 
then <I>A = A'/ 



The last corollary comes from the observation that if a cylindric 
partition 11 is given by a sequence A = (A°, . . . , A"'") then 

(6.8) <^(t)=AA(t)/6Ao(t). 

In the rest of this section we prove generalized MacMahon's formulas 
for skew plane partitions and cylindric partitions that are stated in 
Theorems [7] and [HI The proofs of these theorems were inspired by |Boj . 
[ORj and |V2] . We use a special class of symmetric functions called 
Hall-Littlewood functions. 

6.1. The weight functions. In this subsection we introduce weights 
on sequences of ordinary partitions. For that we use Hall-Littlewood 
symmetric functions P and Q. We recall some of the facts about these 
functions, but for more details see Chapters 111 and VI of |Mac] . We 
follow the notation used there. 

Recall that Hall-Littlewood symmetric functions Px/fj,{x] t) and Qx/^{x 
depend on a parameter t and are indexed by pairs of ordinary parti- 
tions A and /j,. In the case when t = they are equal to ordinary Schur 
functions and in the case when t = — 1 to Schur P and Q functions. 

The relationship between P and Q functions is given by (see (5.4) of 
pacl Chapter HI]) 

(6.9) Qa/m = ^^A/M, 

where b is given with (16. ip . Recall that (by (5.3) of [Mact Chapter HI] 
and 

(6.10) Px/^L = Qx/^, = unless A D /i. 

We set Pa = -Pa/0 and Qx = Q\/%. Recall that ((4.4) of |Macl Chapter 
HI]) 

H{x, y- 1) := Qa(x; t)Px{y; t) = H 

A specialization of an algebra A is an algebra homomorphism p : 
^ ^ C. If p and cr are specializations of the algebra of symmetric 
functions then we write Px/^{p] t), Qx/fj.{p', t) and H{p, a; t) for the im- 
ages of Px/^{x;t), Qx/^{x;t) and H{x,y;t) under p, respectively p®a. 
Every map p : {xi,X2, ■ ■ ■) (ai,a2,...) where G C and only 
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finitely many a^'s are nonzero defines a specialization. These special- 
izations are called evaluations. A multiplication of a specialization p 
by a scalar a G C is defined by its images on power sums: 

Pnia ■ p) = a'^Pnip)- 

If p is a specialization of A where xi = a, X2 = = . . . = then by 
(5.14) and (5.14') of jMiHl Chapter VI] 
(6.11) 

V^A//^(^)a'''^'~'^' A D /i, A//i is a horizontal strip, 

otherwise. 



Q\/t,{p;t) 



Similarly, 
(6.12) 



Px/,{p;t) 



{ip\/fM{t)a^^^ A D /i, A//i is a horizontal strip, 

otherwise. 

Let T > 2 be an integer and = {pf, . . . , p^_i) be finite sequences 
of specializations. For a sequence of partitions A = (A^, . . . , A"^) we set 
the weight function (A; t) to be 

T-l 

w{A) = g^i^'i J2 n P>^-/APn; t)Qxr.^r/Apn; t), 

M n=l 

where q and t are parameters and the sum ranges over all sequences of 
partitions M = {p^, . . . , p^~^). 

We can also define the weights using another set of specializations 
i?± = (i?±, . . . , J were Rf = q^'pf. Then 

iy(A) = ^iy(A,M, /?-,/?+), 

M 

where 

T-l 

iy(A, M, R+) = gl^l II Px«/,n iR-; t)Qx^+i/,n{R+^- 1). 

n=l 

We will focus on two special sums 

^skcw = ^(^) 
A=(0,Ai,...,Ai^,0) 

and 

A=(AO,A1,...,aT) 
aO=A^ 
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6.2. Specializations. We show that for a suitably chosen specializa- 
tions the weight function vanishes for every sequence of ordinary parti- 
tions unless this sequence represents an interlacing sequence, in which 
case it becomes (16.31) . 

Let A~ = {A^, . . . , and y4+ = {Af, . . . , be sequences of 

O's and I's such that + = 1. 

If specializations are evaluations given by 

(6.13) : xi = A^, X2 = X3 = . . . = 

then iy(A) vanishes unless A is an interlacing sequence of profile A~ 
and in that case 

iy(A;t) = $A(t)gl^l. 
Then from Corollaries [3] and H] we have that 

^skcw = Yl ^n(t)gi"i 

ngSkcw(T,A) 

and 

neCyi(T,yi) 

where A~ in both formulas is given by the fixed profile A of skew plane 
partitions, respectively cylindric partitions. 

6.3. Partition functions, li is Xi = s, X2 = = ... = and p~ 

is Xi = r, X2 = X3 = . . . = then 

H{p^,p = . 

1 — sr 

Thus, for specializations given by (16.131) 

+ , _ l + tq^-'A+A- 



(6.14) Hipt,p- 



3 



1 - g^-M+Aj 

We have shown in our earlier paper (Proposition 2.2 of [V2j ) that 
Proposition 6. 

0<i<j<T 

Then this proposition together with (I6.14p implies Theorem [71 The 
generating formula for skew plane partitions can also be seen as the 
generating formula for reverse plane partitions as explained in Intro- 
duction. 

Each skew plane partition can be represented as an infinite sequence 
of ordinary partitions by adding infinitely many empty partitions to 
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the left and right side. That way the profiles become infinite sequences 
of O's and I's. Theorem [7] also gives the generating formula for skew 
plane partitions of infinite profiles A = {. . . , A_i, Aq, Ai, . . . ): 

(6.15) y: Mt)r= n 't^- 

neSkew(yl) i<j 

Ai=0,Aj=l 

Similarly for cylindric partitions, using fl6.14p together with the fol- 
lowing proposition we obtain Theorem [HI 

Proposition 7. 

oo ^ T 

(6.16) z,,,(i?~,i?+) = n n 

n=l ^ k,l=l 

where i^x) is the smallest positive integer such that i = Z(j') mod T. 
Proof. We use 

(6.17) $^gA/M(2;)PA/.(2/) = H{x,y)Y,Qwr{x)P^^/r{y)- 



The proof of this is analogous to the proof of Proposition 5.1 that 
appeared in our earlier paper |Vlj . Also, see Example 26 of Chapter I, 
Section 5 of |Macj . 

The proof of fl6.16p uses the same idea as in the proof of Proposition 
1.1 of [B^. 

We start with the definition of Zcy\{R~ , R~^). We omit index cyl. 

T 

Z{R-,R+) = 5^iy(A,M,i?-,i?+) = 5^gl^inn"-^"(^n)QAVM" 

A,M A,M n=l 

T 

A,M n=l 

If X = {xi,X2, ■ ■ ■ ,xt) is a vector we define the shift as sh(x) = 
(x2, . . . , xr, xi). We set Rq = R^, /iq = /Ut and tq = tt- If using 
the formula (16.171) we substitute sums over A*'s with sums over r*~^'s 



38 SYLVIE CORTEEL', CYRILLE SAVELIEF' AND MIRJANA VULETIC* 

we obtain 



ll,T 



■Qfj.'2/T^iRt)Pfj.^/T^iqR2 ) ■ ■ ■ Q /rT-i{qRT-l)Pfj.T--i /rT'iiRx] 

= i/(gsh/2";i?+)^W^(sh/i,r,gshi?-,i?+) 

= H{qshR-;R-^)Z{qshR-,R^). 
Since sh^ = id if we apply the same trick T times we obtain 

T 

Z{R-,R+) = Yl H {q' sh' R~;R+) ■ Z{q'^R-,R+) 

i=l 
T 

= Y[H{q' sh' R-]R+) ■ Z{sR-,R+). 



i=l 

where s = q^. Thus, 

oo T 

Z =Y\Y\H{q'+^''~^^^sh'R--R+) hm Z{s''R^,R^ 



n=l i=l 

Because 



oo ^ 

hm Z{s'^R',R+) = hm Z (trivial, = TT 

n— »oo n— »oo 1 — 

n=l 

and 

T T 



n H{q'^^^R^,Ri)l[H{q^^'~^R^,R^ 



k=l+l k=l 



l[H{q'sh'R~,R^) = n 

1=1 1=1 

T 
k,l=l 

we conclude that fl6.16p holds. □ 

Observe that if in Theorem [S] we let T — > oo, i.e. the circumference 
of cylinder go to infinity then we reconstruct (16.151) . 

7. Concluding remarks 

In this paper, we determine the generating functions of plane over- 
partitions with several types of constraints. In particular, we can com- 
pute the generating function of plane overpartitions with at most r 
rows and c columns and the generating function of plane overparti- 
tions with entries at most n. The natural question is therefore to put 



PLANE OVERPARTITIONS AND CYLINDRIC PARTITIONS 39 

those constraints together and to compute the generating function of 
plane overpartitions with at most r rows, c columns and entries at most 
n. Unfortunately this generating function is not a product. 
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